Remove `pygw` and checkout `wxflow` #1722

aerorahul · 2023-07-03T03:45:14Z

Description

pygw from the global-workflow has been moved to its own repository at wxflow.
This PR:

removes pygw and replaces it with a checkout of wxflow in checkout.sh and Externals.cfg

Going forward, until wxflow becomes a fixture as a module, global-workflow will be required to be checked out and is done as part of checkout.sh and Externals.cfg

Type of change

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?
CI testing will be used to test this change.
GDASApp tests will also need to be tested. This is unknown.

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

emcbot · 2023-07-03T05:10:38Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Sun Jul  2 22:52:19 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Sun Jul  2 22:54:18 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul  3 00:10:29 CDT 2023
Failed to create experiment:  *FAIL* C48_S2S
Experiment setup: failed at Mon Jul  3 00:10:32 CDT 2023 for experiment C48_S2S

Traceback (most recent call last):
  File "/work2/noaa/stmp/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/work2/noaa/stmp/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96_atm3DVar
Experiment setup: failed at Mon Jul  3 00:10:34 CDT 2023 for experiment C96_atm3DVar

Traceback (most recent call last):
  File "/work2/noaa/stmp/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/work2/noaa/stmp/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96C48_hybatmDA
Experiment setup: failed at Mon Jul  3 00:10:36 CDT 2023 for experiment C96C48_hybatmDA

Traceback (most recent call last):
  File "/work2/noaa/stmp/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/work2/noaa/stmp/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'

emcbot · 2023-07-03T05:10:55Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul  3 03:52:50 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul  3 03:57:19 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul  3 05:10:48 UTC 2023
Failed to create experiment:  *FAIL* C48_S2S
Experiment setup: failed at Mon Jul  3 05:10:50 UTC 2023 for experiment C48_S2S

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96C48_hybatmDA
Experiment setup: failed at Mon Jul  3 05:10:51 UTC 2023 for experiment C96C48_hybatmDA

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96_atm3DVar
Experiment setup: failed at Mon Jul  3 05:10:53 UTC 2023 for experiment C96_atm3DVar

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'

emcbot · 2023-07-03T14:43:55Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul  3 13:32:52 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul  3 13:36:15 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul  3 14:43:48 UTC 2023
Failed to create experiment:  *FAIL* C48_S2S
Experiment setup: failed at Mon Jul  3 14:43:50 UTC 2023 for experiment C48_S2S

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96C48_hybatmDA
Experiment setup: failed at Mon Jul  3 14:43:52 UTC 2023 for experiment C96C48_hybatmDA

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96_atm3DVar
Experiment setup: failed at Mon Jul  3 14:43:53 UTC 2023 for experiment C96_atm3DVar

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'

aerorahul · 2023-07-03T15:16:49Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul  3 13:32:52 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul  3 13:36:15 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul  3 14:43:48 UTC 2023
Failed to create experiment:  *FAIL* C48_S2S
Experiment setup: failed at Mon Jul  3 14:43:50 UTC 2023 for experiment C48_S2S

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96C48_hybatmDA
Experiment setup: failed at Mon Jul  3 14:43:52 UTC 2023 for experiment C96C48_hybatmDA

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'
Failed to create experiment:  *FAIL* C96_atm3DVar
Experiment setup: failed at Mon Jul  3 14:43:53 UTC 2023 for experiment C96_atm3DVar

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/setup_expt.py", line 13, in <module>
    from hosts import Host
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/1722/global-workflow/workflow/hosts.py", line 6, in <module>
    from wxflow.yaml_file import YAMLFile
ModuleNotFoundError: No module named 'wxflow'

I don't think this is a branch issue, but a CI issue.
I have cloned and built and setup the experiment from this branch on Hera and I get no such errors.

cd /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/scratch/global-workflow
./workflow/setup_expt.py gfs forecast-only --app ATM --pslot ccc --idate 2013040100 --edate 2013040100 --gfs_cyc 1 --resdet 384 --expdir /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/EXPDIR --comrot /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/ROTDIR
forecast-only mode treats ICs differently and cannot be staged here
EDITED:  /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/EXPDIR/ccc/config.base as per user input.
EDITED:  /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/EXPDIR/ccc/config.aeroanl as per user input.
EDITED:  /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/EXPDIR/ccc/config.landanl as per user input.
EDITED:  /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/EXPDIR/ccc/config.ocnanal as per user input.

sorc/link_workflow.sh

WalterKolczynski-NOAA

Provisionally approved pending CI testing

emcbot · 2023-07-07T22:45:06Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jul  7 21:32:50 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Jul  7 21:36:59 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Jul  7 22:44:56 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:44:59 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:02 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:05 UTC 2023 for experiment C96_atm3DVar

emcbot · 2023-07-07T23:27:08Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jul  7 21:32:50 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Jul  7 21:36:59 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Jul  7 22:44:56 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:44:59 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:02 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:05 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Fri Jul  7 23:27:07 UTC 2023
with 18 successfully completed jobs

emcbot · 2023-07-08T01:12:12Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jul  7 21:32:50 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Jul  7 21:36:59 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Jul  7 22:44:56 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:44:59 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:02 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:05 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Fri Jul  7 23:27:07 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Sat Jul  8 01:12:11 UTC 2023
with 148 successfully completed jobs

emcbot · 2023-07-08T01:45:15Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jul  7 21:32:50 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Jul  7 21:36:59 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Jul  7 22:44:56 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:44:59 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:02 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:05 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Fri Jul  7 23:27:07 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Sat Jul  8 01:12:11 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Sat Jul  8 01:45:15 UTC 2023
with 86 successfully completed jobs

emcbot · 2023-07-08T01:48:07Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jul  7 21:32:50 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Jul  7 21:36:59 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Jul  7 22:44:56 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:44:59 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:02 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Jul  7 22:45:05 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Fri Jul  7 23:27:07 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Sat Jul  8 01:12:11 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Sat Jul  8 01:45:15 UTC 2023
with 86 successfully completed jobs

aerorahul added 2 commits July 2, 2023 23:31

remove pygw and replace with git submodule of wxflow

07ba4ea

update documentation to reflect checkout with submodules

8d5beb1

aerorahul requested review from WalterKolczynski-NOAA and HenryRWinterbottom July 3, 2023 03:45

remove pytests, update README and ci script

218bdc1

aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Jul 3, 2023

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Jul 3, 2023

update submodules when checkout

7174197

aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Jul 3, 2023

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Jul 3, 2023

add .editorconfig to project

f5cdab4

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Jul 3, 2023

TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jul 3, 2023

aerorahul added 4 commits July 7, 2023 15:21

import directly from wxflow

b19af27

ci: import directly from wxflow

066e14a

scripts/ush/python/pygfs: import directly from wxflow

52b9b3a

remove submodule from ci

232efaa

aerorahul changed the title ~~Remove pygw and replace with git submodule of wxflow~~ Remove pygw and checkoutwxflow Jul 7, 2023

aerorahul changed the title ~~Remove pygw and checkoutwxflow~~ Remove pygw and checkout wxflow Jul 7, 2023

update docs to remove referecnes to submodule

d431f6a

aerorahul reopened this Jul 7, 2023

WalterKolczynski-NOAA removed CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Jul 7, 2023

WalterKolczynski-NOAA requested changes Jul 7, 2023

View reviewed changes

sorc/link_workflow.sh Show resolved Hide resolved

aerorahul added 2 commits July 7, 2023 16:59

fix pycodestyle errors

3981747

link and copy business

9126d55

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Jul 7, 2023

WalterKolczynski-NOAA approved these changes Jul 7, 2023

View reviewed changes

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Jul 7, 2023

put back tic tic tic in ci script

3fe6f4e

emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Jul 7, 2023

Merge branch 'develop' into feature/wxflow

a447d1e

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jul 8, 2023

WalterKolczynski-NOAA merged commit 4fdf144 into NOAA-EMC:develop Jul 10, 2023
4 checks passed

aerorahul deleted the feature/wxflow branch March 19, 2024 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove `pygw` and checkout `wxflow` #1722

Remove `pygw` and checkout `wxflow` #1722

aerorahul commented Jul 3, 2023 •

edited

Loading

emcbot commented Jul 3, 2023

emcbot commented Jul 3, 2023

emcbot commented Jul 3, 2023

aerorahul commented Jul 3, 2023 •

edited

Loading

WalterKolczynski-NOAA left a comment

emcbot commented Jul 7, 2023

emcbot commented Jul 7, 2023

emcbot commented Jul 8, 2023

emcbot commented Jul 8, 2023

emcbot commented Jul 8, 2023

Remove pygw and checkout wxflow #1722

Remove pygw and checkout wxflow #1722

Conversation

aerorahul commented Jul 3, 2023 • edited Loading

emcbot commented Jul 3, 2023

emcbot commented Jul 3, 2023

emcbot commented Jul 3, 2023

aerorahul commented Jul 3, 2023 • edited Loading

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

emcbot commented Jul 7, 2023

emcbot commented Jul 7, 2023

emcbot commented Jul 8, 2023

emcbot commented Jul 8, 2023

emcbot commented Jul 8, 2023

Remove `pygw` and checkout `wxflow` #1722

Remove `pygw` and checkout `wxflow` #1722

aerorahul commented Jul 3, 2023 •

edited

Loading

aerorahul commented Jul 3, 2023 •

edited

Loading